Recently, as smartphones and digital still cameras become more and more affordable, almost everybody has one such device. When traveling, therefore, people tend to take pictures of the beauty in sight with the smartphones or digital still cameras they carry with them as a souvenir to remember the scenery by. In particular, many new models of smartphones and digital still cameras are provided with a “panorama function” for taking panoramic photos. A user only has to switch such a smartphone or digital still camera to the “panorama mode” and perform a “horizontal scanning” operation, taking images of different parts of the same site, and a panorama will be produced by a stitching operation of the image processing software installed on the smartphone or digital still camera, allowing the user to faithfully record the grandeur of a magnificent view.
Generally, software designed to produce a panorama by stitching together at least two images taken respectively of different parts of the same site—be it image processing software pre-installed on a smartphone or digital still camera, or commercially available image editing software to be installed on an electronic device (e.g., a computer)—includes the following stitching procedure to be performed by the smartphone, digital still camera, computer, or like electronic device (hereinafter collectively referred to as the electronic device):
(1) It must be understood in the first place that a panorama is in fact an assembly of a number of images. To start with, a plurality of images are taken horizontally and consecutively of different parts of the same site with a smartphone or digital still camera. Take a common smartphone or digital still camera for example. The view angle of the lens to be used is typically 60 degrees, so at least eight images must be taken in order to produce a 360-degree panorama. Basically, the more images are stitched together, the better the result. When producing a 360-degree panorama, referring to FIG. 1, it is crucial that each constituent image has an appropriate portion reserved as an “overlap area”, and the larger the overlap area, the better the stitching result. This is especially true when a lens of a relatively short focal length (e.g., 18 mm, 28 mm, etc.) is used, for it is difficult to perfectly stitch together images with a strong sense of perspective, and a more natural stitching result can be obtained in that case by taking more images and making the overlap area as large as possible. To ensure that a 360-degree panorama is attainable, therefore, it is common practice to take more than ten images horizontally and consecutively of different parts of the same site using a smartphone or digital still camera.
(2) The electronic device sequentially adjusts the brightness and hues of each image with the image processing or editing software installed thereon, in order to achieve consistency in brightness and hue.
(3) Then, the electronic device stitches together the overlap areas of each two adjacent images with the image processing or editing software.
(4) Once the overlap areas are stitched together and fused, the user has to trim the four sides of the resulting image with the cutting tool of the image processing or editing software in order to complete the panorama.
When the electronic device performs the aforesaid stitching and fusing operation on at least two real-site images, errors of the image registration algorithm or parallax in the real-site images tends to hinder perfect registration of the images in the corresponding overlap areas, or stitched areas, of the real-site images, resulting in misaligned images, ghost images, or similar defects in the panorama obtained. While it may be acceptable to have a misaligned image or ghost image in areas that are not intended to be the focus of attention, the overall visual effect of the panorama will be seriously impaired if such a defect takes place where viewers' attention is intended or inevitable (e.g., on a human face, human figure, flower, car, building, or other visual target in which a viewer may take interest).
The following paragraphs provide a detailed description of how commercially available image processing or editing software stitches at least two real-site images together:
(1) First, the electronic device to which the software is applied reads the at least two real-site images and chooses a stitching path for the overlap areas of the real-site images according to the image processing or editing software. If a linear fusion/stitching algorithm is used, referring to FIG. 2, in which images IA and IB are registered overlap-area images of the two real-site images, the process of choosing the stitching path begins by calculating the difference diagram ΔI=abs (IA−IB) of the two overlap-area images IA and IB.
(2) Then, a minimum-difference path S in the difference diagram is found using a dynamic programming or graph cut algorithm. More specifically, all the pixels on each of a plurality of different paths in the difference diagram ΔI are added up to obtain the difference of each path, and the minimum-difference path S is determined by comparing the differences of the paths. The minimum-difference path S is defined as the path with the smallest difference of all the different paths that run through the overlap-area images IA and IB.
(3) Following that, a mask diagram Imask is constructed, the process of which basically involves assigning the value 0.5 to each pixel on the minimum-difference path S such that the farther a pixel in the difference diagram ΔI and to the right of the minimum-difference path S is from the minimum-difference path S, the greater (or the closer to 1.0) the pixel's value, and the farther a pixel in the difference diagram ΔI and to the left of the minimum-difference path S is from the minimum-difference path S, the smaller (or the closer to 0.0) the pixel's value.
(4) Once the mask diagram Imask is obtained, the overlap-area images IA and IB are fused using a linear fusion algorithm or a Laplace pyramid fusion algorithm. The linear fusion algorithm is the simpler of the two and is carried out by weighting the to-be-fused overlap-area images IA and IB by the mask diagram Imask and then performing linear superimposition to form a fused image Iblend=IA(I−Imask)+IBImask. Please note that the minimum-difference path S obtained is also referred to as the “stitching path”.
Since the foregoing stitching operation does not take into account whether or not the overlap-area images IA and IB respectively contain images of a portion, or the entirety, of a common subject that compels attention (e.g., a human face, human figure, flower, car, building, or like visual target), the minimum-difference path S obtained is very likely to run through the subject. If, for example, the subject is a human face as shown in FIG. 3, in which Ia and Ib are registered overlap-area images of the two real-site images, the stitching path (or minimum-difference path, as indicated by the zigzag line on the right of FIG. 3) in the difference diagram ΔIab of Ia and Ib may pass through the face such that, upon completion of the image stitching and fusing process (see FIG. 4) and due to registration errors between the images Ia and Ib, a ghost image can be seen on the nose (as shown in the dashed-line circle on the right of FIG. 4), where the stitching path traverses. Thus, the subject of interest or focus of attention (i.e., the face) in the resulting panorama is defective in terms of visual effect, which compromises the image quality of the panorama significantly.
Moreover, referring to FIG. 5, in cases where the real-site images have relatively small overlap areas Im and In but the image of the to-be-protected subject (e.g., the image of a human figure) occupies a major portion of each overlap area, it is extremely difficult for the minimum-difference path S determined by either of the aforesaid minimum-difference path algorithms to bypass the image of the subject (e.g., the image of the human figure). If the overlap areas Im and In are directly stitched together and fused after whole-area registration, there must be misaligned images or ghost images in a local area of interest (e.g., a human face) of the fused image Iblend as a result of matching errors, parallax, or the like, thus substantially lowering the image quality of the panorama.
According to the above, it is imperative to protect the subject of interest in the overlap areas of the real-site images to be stitched together and fused. The issue to be addressed by the present invention, therefore, is to design an image stitching method for producing a panorama and to ensure that, when an electronic device applies the method and stitches at least two real-site images together to produce a panorama, the images of a common subject in the overlap areas of the real-site images will be perfectly stitched together.